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What  I  really  want.. 

•  My  personal  information  to  be  accessible 
0  Publistied  information  should  find  me 

•  Usable  anywhere 

•  Others  can  use  what  I  have  learned 
(if  I  want  them  to) 
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What  is  it? 


Electronic  Publishing 

(Or  publishing  over  wires) 
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New  Communications  Technology  Problems 

BOOKS 

Telegraph> 
Telephone 

Electronic 
Publishing 

Experts  only 

Monks 

Operators 

Professional 
searchers 

Distribution  Is  hard 
and  expensive 

Vellum  is 
calfskin 

Telephones  on 
barb  wire 

$1/mlnute  over 
obscure  modems 

Different  interfaces 

1000's  of  languages 
in  Europe  alone 

Switching  was 
manual 

//query  (W5) 
inform? 

Material  Is 
intractable 

Scrolls  and  manu- 
scripts were  about 
as  random  access 
as  musical  scores 

No  white  pages 

600  databases 
on  Dialog 
~1  Terabyte 
140Gbyte  at  DJ 
80GB  card  catalog 
atRLG 

Business  model 
needed 

Centralized  printing 

Pay  per 

minlltf 

Not  understood 

1 

Wide 
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Navigation  Techniques:  Paper 

«  Alphabetical  Listings  (dictionary,  Encyclopedia) 
0  Indices  (back  of  the  book  and  Readers  Guide) 
«  Table  of  Contents  (outlining) 

•  Citation  index 

•  "Tree  of  Knowledge" 

«  Have  you  read  any  good  books  lately? 
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Navigation  Tecliniques:  Computers 

•  Hierarchical  File  Systems 

•  Unix  "find"  and  "grep",  Mac  "find  file" 

•  Gopher,  Magellan,  ON  Location 

•  Boolean  query  systems  (...within  5  words  of...) 

•  Static  Hypertext  links  (see  also  pointers) 
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Navigation  Techniques:  WAIS 


•  English  language  questions  and  relevance 
feedback 

-  Question-answer  dialog 

-  Similar  to  Newspapers:  "More  on  page  5" 

-  Dynamic  Hypertext  Links 

•  2  level  search: 

-  Directory  of  servers  (server  like  any  other) 
Servers  themselves 
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Wide  Area  Information  Server 
Architecture 


Directory  of 
Servers 


Gateways 
to  other  nets 


Z39.50  over 

X.25,  TCP/IP,  Modem 

Open  Interconnection 

Public  Protocol 


Z39. 
over 
LAN 


^E] 


5C- 


E] 


LE] 


TV  Guide 
etc. 


Users  Needs: 

Selecting  Servers 
Answering  Questions 
Organizing  Responses 


Architecture  Issues: 

Scalability 

Security 

Business  model  for  servers 

Reliable  Access 
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Demonstration  System  Structure 

WAIS 

LAN                           V^ 

Dow  Jones 

pHffl 

"*  Workstations 

«  •  • 

X.25       gg 

o£,¥°. "-«-    U-*4Bi 

9600Baud       Machine                        KJ 

Operations: 

Archiving 

Queries 

Retrieval 
IRType: 

Broadcast 

Query  by  Example 
Databases: 

Wall  St  Journal 

Barron's 

400  Business  Mags 

CM: 
Operations: 

Queries 
IRType: 

enhanced  relevance 

feedback 
Databases: 

DowVision  and  memo's, 

mail,  word  processor  files 

Mac: 
Operations: 

Human  Int 

Retrieval 

Queries 

'Caching"  Docs 

User  Profiles 
IRType: 

Query  by  example 
Databases: 

Personal  Text 

Cached  data 
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WAIS  Clients 

•  Busy  24  hours  a  day  finding  information 

•  Ponder  all  indications  of  the  preferences  of  its  user 
«  Gossip  with  other  clients  about  their  discoveries 

•  Scours  the  world  (within  a  budget)  to  find 
new  sources 

•  Current  implementations  on  PC,  Macintosh, 
X  Windows,  NeXT,  dumb  terminal  (dial-up) 
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WAIS  Protocol 

•  Based  on  NISO  Z39.50  international  standard 

•  Flexible  —  separates  clients  from  servers 

•  Search:  (words,  docjds,  databases) 
returns  list  of:  (headline,  score,  docjd,  types) 

•  Retrieval:  (docJd,  type,  start,  end)  returns: 
data  of  specified  type 

•  DocJd:  An  ISBN  for  the  Electronic  Age 

•  Server  Description  Structure  for  the  Directory 
of  Servers 
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How  standard  Protocol 
can  Provides  Security 

Users  do  not  login  to  server,  but  search  only 
through  application  layer  protocol  (Z39.50). 

Server  controls  access  to  data. 

Network  layers  below  application,  or  application 
layer  handles  authentication,  encryption,  billing. 
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The  WAIS  Protocol  /s  WAIS 


Supports  any  search  syntax 

Supports  sophisticated  clients  —  puts 

intelligence  in  the  user's  hands 

Clients  can  run  on  any  platform 

Multiple  servers  in  a  single  search 

Retrieve  any  kind  of  data:  text,  graphics,  video,... 
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Connection  Machine  Server 

1-100GBytes  (and  getting  bigger) 

Supports  thousands  of  users 

Automatic  Indexing 

Uses  words  and  phrases  in  question  to  find 
appropriate  documents  with  relevance  feedback, 
weighted  term 

Supports  Boolean  Queries 

Cost  effective  hardware  alternative  to  mainframes 
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Unix  Based  Server 

1MB-10GBytes 

Supports  hundreds  of  users 

Multimedia  indexing:  word  processor  files  etc. 

Supports  Natural  language  and  Boolean  Queries 

Easy  to  install 
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Results  Improve  with  Query  Size 

3.\ 


Precision  x  recall 
@  25%  recall 


Average 
performance 
over  13 
reference  sets 


3. 
A 

A 


t 


Typical 
Relevance 
Feedback 
Query 


Typical 

Boolean 

Query 


'WWWWWWWWW^ 

number  of  query  terms 
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WAIStation: 

Active  Database  Sources, 

Saved  Questions 


oi 


i  Sources  I 


<S>  CM  spplwa^ions 

<S>  King  J  jm«s  Bible 
<S>  Macintosh  Hard  Disk 
<9>  TMC  Business  tmail 
<S>  TMC  Library  Catalog 
«>  fi<i//Sf.  <Awrnj/ 
<S>  World  Faotbook 


O 


o 


Questions            | 

?    CM  Apps  Ou*stion 

7    Library  question 

?    Encgolopedia  0 

?    PaUntO 

?    TMC  Buf .  Email  0 

?    TMCFunO 

?     Montvalt  Q 

?    World  Faotbool<  g 

?    poetry  q 

?    Bible  0 
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Select  Data  Source 

sn==7  Sources  ===| 

<S>  CM  apptioations 
<S>  Encyclopedh 
<S>  King  James  Bibl* 
<»  Macintosh  Hard  OlsH 
O"  TMC  Business  *mail 
<«>  TMC  Libr^i45ji.^::$^:$ 

0: 

1  Run  1 

Vhioh  are  sim.'T  to  In  the5«  soirees 

1           w 

•«>  V-lifSf.JbiOnj/ 

5 

<a>  Vorld  Faotbook 

Results 

O: 
2 

Q 

B 

V\ 

MeAr 

3a 

nformation 
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Run  Initial  Queiy 


i  Questlon- 


Look  for  daeumeMs  ahaut 


recent  developments  1n  personal 
computersj 


1^ 


Vhich  ar<?  similar  to  In  thesg  sources 


<S>  k<3//St  <Awrvj/ 


Results 


g)  ***  Compaq  Computer  Db-ectors  Approve  2-for-1  Stock  Split  Q^ 
g|  *ir*  International :  Bull  Agrees  to  Pay  Zenith  $15  Million  to  En*!""^ 
1=1  **■*  AT&T  Set  to  Announce  Memorex  Computer  Accord 
Hi  ***  Technology  Brief-- fciternational  Business  Machines  :Pri< 
m  ***  Busfriess  Brief  "  Data  General  Corp.:  Foiir  Models  Are  Uf 


[?)  ***  Retailing:  Businessland  Enters  Japan,  Aided  bg  4  Big  looa 


m 
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lick  a  Headline  to  Display  a  Document 


L»tk  f»r  <»e»n»«><t  ifct«rt 


recsnl  developmenis  In  personal 
compulerj 


SQ 


Vfctah  art  sl-nar  !■  I»  thcw 


In 


*  Corrpjqtanpulff-Olredori  A(proTe2-f(r-1  Stodk  Spill 

*  htmMt<tinal:BunA?-*<sloPavI«iiHh|tSm!hi>ila'- 
>  AT&T  &<t  U  Annauriw  Htfnci^vCttrpul«r  Aa«a-( 

.  T<c>in»Vg:j  Ek-ltf—^Urntll«u1Bufin«tt  M(ehl%ti:PHi 
.  BmhwiETlff— P»>GHWfilCon).;F{'trrWtli  ' 


■  R4l 


ais 


'^  ^  Taehnntagy:  Computer  FIrmtSBB  the  lUrltlngi" 


InUrMtlonal  Btulnest  ti«hlMjCorp.,  AppU  Umputtr  fnc. 
end  other  trig  compultr  rmken  are  jtaM  ng  «ut  pijitlenj  In 
thsnMMnlmarUl  for  "mts- pad  ««Mp«ttri," small  rnschiMa 
thiUetuMr»<M!rdft((tuwr1tlnjrstberth)(iUf>plri9 
liaija.The  nct«  p«d3  Igplratlu  reccgnia  BiimtNrBaivllBtlera 
pHftte4onaKrMnvltht«pM(tilpenaE>dcenver(thtm1nt« 
«rivintl>ni1  lUclronlc  cMrKttn.Thi  thrarmaManli  tMn 
atarBdfsrUUrtrBnsrflrtaepirsiBalcoinpuUrDri 
company')  melnnnpilcrs. 

The  ifM  of  IM  mirfcrf  for  noli-pidCiBpiUra  1»n'l  tUir, 
l)irtlnfcKrpjaSant«Cl9r8jC«llf„BTsrkBt-rese9fetiftrn, 
ut1m»le»  tM  marlcftt  will  grov  ta  3.4mllllQR  unlu  lotd  In 
1 995  from  22,000  Urtts  Itils  «{Br.  Only  w«  wir^anv,  Torrig 
Corp.'a  GridaysUma  ufiH.eiirrently  aelU  rote-p*l  canputira 
In  Ihe  U.S.J  IUf(*Je!,iMrodwe<ll»t  September,  ttprl«J 
at  J3,aa0.  Bill  new  vanlitreatren'pecledla  Introduce  eeverg) 
n»ti-p«d  mieM  nw  f  Ms  iittr.  And  tlretdy,  big  wmputtr  mikirt 
era  fighting  quietly  for  cunirol  ovar  aoftvere  standards  for 
thill  ftdgtttj  vhkh  r»qulr»dlfrir4ril  pr^grimt  froKi  thow 
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Relevance  feedback". 
Find  me  more  like  this  one" 


1  Question-l 


Look  for  documgnts  about 


recent  developments  in  personal 
computers 


Vhich  ar*  similar  to  In  Ihtse  soaros 


|£|  Technology :  Coi 


<S>  i/j//Sf.  wOwtm/ 


O 


«**  Compa<i  Computer  Directors  Approve  2-for-1  Stock  Split 
***  International :  Bull  Agrees  to  Pag  Zenith  $i5  Million  to  P"' 
«**  AT&T  Set  to  Announce  Memorex  Computer  Accord 
«**  Technology  Brief—  International  Business  Machines: 
1,**  Business  Brief--  Data  Oeneral  Corp.:  Four  Mo(iels  Are  Ur 

B  *«•  Relailingl  BusfiMsslind  Enttrs  Japan,  AWed  Im  4  Big  Uoa 

r\    .  .  .   r- n^^^  s^  flmnHfiftatinnr 


EHiB 

ii 

Prt.  ■ 
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Relevance  Feedback  of  Paragraph 


Technologii:  Computer  Firms  See  the  IMrlting  i 


h  in  on  people  who 


Computer  makere  are  ecrambli  nq  to  ( 
find  the  pen  mlgMier  than  the  keijboarc). 

I nternational  Busi ne(J  Mtchi nea  Corp.,  Apple  Computer  Inc. 
^nd  other  tigcompoler  maliers  are  staking  out  poaittons  in 
Se  nascent  market  for  "note-pad  computers,"  small  machines 

fet  let  oeera  enter  data  by  vritinq  rather  than  tapping 

lis.  The  note  pads  typically  reajgnize  numbers  and  letters 
Sited  one  acreen  vith  a  apecial  pen  and  convert  them  i  nto 


I  QuBstion-l 


look  for  documtnts  aboirt 


ecent  devalopmsnts  in  personal 
omputers 


is 


^hicli  ar*  similar  to  in  thgse  sources 


ST(ohnoto9y:CijiO     tt  nnst.iMrml 


O 


RaguUfi 


{PJ  ***  Conpaq  Computer  Directors  Approve  2-for-l  Stock  Split 
I?)  ***  International:  Bull  Agrees  to  Pay  Ztnllh  $15  Million  to  En 
(?|  •**  AT&T  Set  to  Announce  Memorex  Computer  Accord 
i)  «**  Technology  Brief  ~  International  Business  Machines  :Pri* 
[g|  *•*  Bushess  Brief  —  Data  General  Corp. :  Four  M«d«ls  hr^  Ur 
^  ***  Teohnolow  '•  Computer  fVws  See  the  Vriting  on  the  Seres 
|?1  ***  Retailing;  BustiessUnd  Enters  Japan,  Aided  bg  4  Big  Loca 
i;^    ...^  r-. n^,.^A   tr^^llft- .tinny 
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it 


airiing"  of  Questions  to  Follow  a  Tangent 


jqqk  for  Jocummlg  about 


Irecent  developments  In  personol 

computers! 


I  m-ni 


Vhloh  Jr»  similar  In  l«  tfces«  souroes 


®  YiHSt.JMmti 


.  hUrMttefiil:  8ull  Agrwj  to  Pig  Henith  $1 5  Mllllor.  to  En*0 
k  AT&T  3«t  tQ  Amounc*  Htmorvx  Cnrvultr  AwonJ 
k  Ttchnok>9ij  Bri«f"  Inttrnitftxial  Busfnn)  M«chh«f  ;Prii 
*  BusimsfBritf  "0»t»C«rrtr»ICorp.:Fo«-MwJ»U  Ar»  Ut 
»  TTctmoktM :  Cwrguttr  FJrmt  St»  Ih*  Vritkni  en  th»  3crn 


Heanlla 


|?|  ***  R«tii11n9:Bui<iMn)}ndEnWf  •Jipinj  Aldtd  by  4Bi9LDM^ 

111  ♦♦  Vh»l'«  H«ws  "  Busirms  and  Fininw 

13  **  Tichnokgy ;  Computer  M*erf  Aqrw  on  «  StjmJird  For  I 

[fl  *  b^«Tr*ck;Busin«sf1tnd[>WotcirETA«  iLon  AndTra 

[^  *  Ttehnobgy  &H»a1th:Bujtn»ss1widToR«pci<-ILo»Ftt-3r- 

E  •  Teehftotogii :  U5.  Cwnpulw  M*k»r  T*l<«  on  NEC  on  Hs  Ow 

B  •  TsdiwlDgy :  Cotr^uln'  Finns  S»  ih9  Vrillng  on  lh»  S«r«« 


JJa 
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Internet  Freeware  Release 

•  Example  User  interfaces  for  free  (no  support) 
Macintosh,  Gnu  Emacs,  Xwindows 

•  Example  unix  server  software  to  create  servers 

•  Directory  of  Servers  on  the  internet  at  least 
through  "92 

«  300  Servers  now:  Weather  Maps,  patents,  journal 
abstracts,  email  archives,  usenet  recipies,... 

•  Free  Software  via  FTP  from  Think.com:/wais/* 

Mailing  list:  wais-discussion-request@think.com 
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WAIS  Uses 

Over  15,000  users  on  the  Internet 

Users  in  28  Countries:  Mexico,  Singapore, 
Finland,  Australia,  etc 

300  Databases  served  from  12  Countries: 
Norway,  Canada,  UK,  etc.  Average  3  new 
databases  registered  per  weelc. 
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WAIS  Uses: 
Campus  Wide  Info  Servers 

•  Class  catalog  and  schedule 

•  Campus  events:  movies,  sports 

•  Job  listings 

•  Library  catalog 

•  Phonebook 

•  Professor  research  interests 

•  Past  theses 

[      sol.acs.unt.edu]   UNTComputerDoc 
[       xantos.ulo.no]   UiO_Publications 
[    next2.oit.unc.edu]   ibm.pc.FAQ 
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WAIS  Uses:  Libraries 

•  Easy  to  use  card  catalog 

•  Remote  use  from  home  or  office 

•  Pictures,  full  text,  scanned  documents 


[pegun. law. Columbia. e]  columbia-law-library-catalog 
[pegun. law. Columbia. e]  columbia-spanish-law-catalog 
[      quake.think.com]   tmc-library 
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WAIS  Uses:  Biology 

0  Journal  Abstracts 

•  Sequence  archives 

•  Images 

Currently  over  20  Biology  databases  in 
Finland,  Netherlands,  and  US 


cmns .think . com] 

bio.vu.nl] 

genbank . bio . net ] 

wais. funet.fi] 

wais.funet. fi] 

wais.funet.fi] 


Molecular-biology 
biology-compounds 
biology- journal-contents 
bionic-ai-researchers 
bionic-directory-of -servers 
bionic-enzyme 
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WAIS  Uses:  Chemistry 
CORE  Project 

•  All  published  chemistry  (8  years  all  ACS) 
»  Scanned  pictures,  ascii  text 

•  Optical  jukebox  mass  storage 

«  Connection  Machine  /  Newton  search  engines 

Project  of  Bellcore,  ACS,  Chem  Abstracts, 
OCLC,  Cornell,  and  Thinking  Machines 

[   cujo.curtin.edu.au]   chem-eng-current-contents 
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Business  Executives 

•  Dow  Jones  information 

•  Corporate  information 

•  Personal  information 

Project: 
KPMG,  Apple,  Thinking  Machines,  Dow  Jones 


[  dims. think. com]      wall-street- journal-sample 

[  think.com]      Business-email 
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WAIS  Uses: 
Medical  Researchers/Doctors 

Medical  papers 

Storing  and  matciiing  patient  records 

Remote  connections  to  specialized  databases 

r  wais.funet.fi]      bionic-databases-limb 
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WAIS  Uses: 
Community  Information 

•  Dial-up  users:  no  network  required 

•  Directories  of  services  or  facilities 

•  Education  and  entertainment 


[  quake. think. com]  internet-resource-guide 

[  sol.acs.unt.edu]  online-libraries 

[  quake.think.com]  weather 

[  lambada.oit.unc.edu]  nsf-bulletins 
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Conclusion 


•  Electronic  Publishing  can  fill  niches  now. 

•  WAIS  Inc  is  working  with  groups  that  want  to 

publish  their  information  internally  or 
externally. 

•  WAIS  Inc  is  working  with  those  that  want  to 

profit  by  using  other  peoples  servers. 
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WAIS  Hardware  Components 


Ccnnedion  Ml  thine 
CMla 


Macintosh  ruYiing 
WAlStaHonvia  MacTCP 


Workstation  rum'ng 
WAIS  via  X- Windows 
orGMACS 
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Data  Parallelism: 

Searching  all  the  Documents  at  Once 


Pharmaceutical  +12 
FDA +9 
Medical  +6 


Stadium 
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Boolean  Search 


Retrieve  documents  containing 
specific  combinations  of  words 


Conceptual  Search 


Explore  a  set  of  documents 
containing  related  concepts 
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Boolean  Query 


Hard  to  Use: 
Complex  Syntax 


(Japanese  OR  Japan)  AND 

(building  OR  buitdings  OR  (Real  AND  Estate)  AND 

(Manhattan  OR  (New  AND  York) 


Poor  Results: 

The  wrong  information 

No  ranlcing  of  results 


Have  you  been  paying  attention?... 
Freer  Finance:  U.S.  Regulators  Move... 
REAL  ESTATE:  California  Initiatives. 
First  Boston  Said  To  Agree  on  Sale  Of.. 
Exxon,  Rockefeller  Group  to  Sell  Site... 
What's  News-Business  and  Finance 


Wide  Area  Information  Servers 


WAIS  Inc. 


Conceptual  Search:  Phase  1 


Easy  to  Use: 
No  Syntax 

Options: 

Wiiat  do  you  want 
to  follow  up? 


Japanese  buying  real  estate  in  mid-town  manhattan 


1.  Time  Acts  to  Cut  Magazine  Costs... 

2.  First  Boston  Said  To  Agree  on  Sale. . . 

3.  Have  You  Been  Paying  Attention? 

4.  Exxon,  Rockefeller  Group  to  Sell  Site... 

5.  Hard  Sell:  Real  Estate  Developers... 

6.  What's  News-Business  and  Finance... 

7.  Integrated  Resources  Buys  Loft  Building... 
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Concsptual  Search:  Phase  2 


Relevance  Feedback: 
I  like  these;  show 
me  more. 


Improved  results: 

Articles  on  related 
topics  are  found. 

Results  are  ranked. 


First  Boston  Siiicl  To  Agree  on  Sale... 
Exvon,  Rockefeller  Group  to  Sell  Site. 


1.  Bids  for  Exxon  Building  in  New  York... 

2.  Time  Acts  to  Cut  Mngii/.iiie  Costs... 

3.  Hard  Sell:  Real  Estate  Developers... 

4.  Time  Inc.  Sells  Its  45%  Interest... 

5.  Citicorp  Unit  Mo^es  to  Foreclose  on... 

6.  Litigious  Landlords:  Legal  Maneuvers 
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Query  Broadcast  To  Database 
on  Connection  Machine  System 


Document  Units       Scores 


75 


Tripoli 


Libyan 


PLO 


bomb 


42 


80 


52 


17 
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Document  Retrieval  Performance 

•    Current  algorithm  limits: 
~2GBwith512MBCIVI-2 
~8  GB  with  2  GB  CM-2 
-25  GB  with  8  GB  CM-2 

High  recall  •%  Stanfill  and  Kahle, 

see    Communications  of  ttie  ACM , 


) 


•  High  precision    J  December  1986 

•  «  1  sec.  response 

•  Much  larger  DBs  searchable  with  CM-5 

and  inverted  index  algorithms:  100s  to  1000s  of 
Gigabytes 
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